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There is an increasing requirement from high levels 
within the Government that the Navy's aircraft cost 
estimators and analysts provide explicit estimates for the 
sub-elements of Aircraft System Test and Evaluation (AST&E) 
efforts. The data required to produce more accurate and 
detailed estimates represent lower levels in the Aircraft 
Work Breakdown Structure (WBS) than previously available. 
This is a two volume thesis. Volume I examines the WBS and 
Contractor Cost Data Reporting (CCDR) system with a 
description of current reporting practices and implement- 
ation shortcomings. Recommended courses of action to 
improve reporting requirements and thereby improve data 
quality and cost estimates are proposed. Major cost drivers 
for AST&E, from both the perspective of Defense Contractors 
and Military Flight Test Centers, are discussed. Beginning 
in Volume II, a relational data base system is introduced to 
more easily evaluate AST&E cost elements and physical/ 
performance characteristics. A Contractor Flight Test cost 
estimating relationship (CER) is developed through step-wise 
multiple regression analysis of data gathered from Defense 
Contractors and Naval Air Systems Command (NAVAIR) . 
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I . INTRODUCTION 



A. PURPOSE OF PROJECT 

There is an increasing requirement from high levels 
within the Government that the Navy’s aircraft cost es- 
timators and analysts provide explicit estimates for the 
sub-elements of aircraft system test and evaluation efforts. 
These requests ask for additional data, which represent 
lower levels in the Aircraft Work Breakdown Structure (WBS) 
than previously required, and the methods necessary to 
produce them are needed in order to estimate Aircraft System 
Test and Evaluation costs in greater detail and with better 
accuracy than is currently feasible. 

The ultimate objective of the Naval Air Systems Command 
(NAVAIR) is to develop cost estimating relationships for all 
of the following System Test and Evaluation elements: 

1. Wind Tunnel Article and Test 

2. Static Article and Test 

3. Fatigue Article and Test 

4. Drop Article and Test 

5. Contractor Flight Tests 

6 . Flight Test Instrumentation 

7. Miscellaneous Ground Tests 

8. Support of Contractor Flight Tests 
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9. Navy Technical Evaluation 

10. Operational Evaluation 

11. Contractor Support of Navy Test. 

The necessary tasks to attain this objective are to: 

1. Identify physical, performance, and programmatic 
parameters which are the primary drivers of each of 
the major functional cost categories (engineering 
labor, manufacturing labor, quality control labor, and 
tooling labor) for each of the major elements of 
Aircraft System Test and Evaluation 

2. Develop parametric cost estimation relationships for 
each of the above System Test and Evaluation elements 
and each appropriate functional cost category by: 

a. Defining a sample of pertinent aircraft 
programs 

b. Formulating a work breakdown structure/ 
functional cost element matrix 

c. Acquiring historical cost data 

d. Organizing the data for analysis 

e. Employing statistical methods to develop cost 
estimation relationships, and 

f. Documenting data, sources, rationale, and 
methodology. 

Given the time constraints, attempting to concentrate 
on all eleven elements simultaneously would prove to be both 
unmanageable and inefficient. Therefore, the project 
sponsor has directed that the initial research focus on the 
area of Contractor Test Flights. If conditions permit. Wind 
Tunnel Tests, Static Tests, and Fatigue Tests, could be 
included . 
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B. IMPORTANCE OF THIS STUDY 



The importance of this study is justified by the 
following reasons. First, due to the rapidly increasing 
cost of new aircraft systems in a budgetarily constrained 
environment, the accuracy of cost analysis performed by 
NAVAIR becomes imperative. Second, an integral study of the 
cost problem would provide more insight to NAVAIR to answer 
the ever increasing informational needs that originate as 
high as Congressional Budget Committees and Executive Branch 
Agencies. More important, it is crucial for cost analysts 
to take into consideration new cost drivers, particularly in 
the area of avionics. Also, it is felt that better coor- 
dination with contractors via an improved implementation of 
the Contractor Cost Data Reporting (CCDR) system, WBS 
format, and cost estimation model will result in the 
standardization of reporting procedures across the industry 
and will increase the likelihood of contractor delivery of 
guaranteed performances for allocated funds. 

C. ORGANIZATION OF THIS STUDY 

This is a two volume research study. Volume I is 
organized as follows: Chapter II provides an overview of 
the Contractor Cost Data Reporting (CCDR) system and the 
Military Standard Work Breakdown Structure (WBS). A 
critical analysis of the current cost estimation system and 
its shortcomings is the focus of Chapter III. It proposes a 
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set of feasible courses of action that could be implemented 
to improve the present system. Chapter IV outlines the 
various steps taken to accomplish the proposed research. 
Specifically, a three-step approach is undertaken; 
exploration, analysis, and refinement. Such a study helps 
identify research processes and keys upon which this study 
should concentrate. These issues are analyzed in detail in 
Chapter V. Results obtained from an extensive survey of 
experts in the contractors' System Test and Evaluation field 
are reported. In particular, major cost drivers that are 
frequently identified include aircraft weight, speed, 
avionics, software, and management practices and policies. 

Volume II focuses on the analysis of data discussed in 
Volume I. Chapter I describes the available current data 
and its structure. Current parametric techniques are 
surveyed in Chapter II. A requirements analysis, design, 
and development of a data base management system called 
TIGER is performed in Chapter III. The purpose of the data 
base is threefold. First, it provides immediate and 
precise answers to ad hoc queries that cost analysts could 
pose. Second, it outlines a well-structured basis for 
standardizing the accounting process handled by various 
contractors. Third, it can be used to assist statistical 
analysis for cost estimation. The data collected is 
analyzed, discussed and cost driver models are developed and 
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presented in Chapter IV. Finally, conclusions derived as a 
result of this study are discussed in Chapter V. 



13 



II. DEPARTMENT OF DEFENSE REQUIREMENTS 
FOR CONTRACTOR COST DATA REPORTING 



A. THE CONTRACTOR COST DATA REPORTING (CCDR) SYSTEM 
1 . Introduction to CCDR 

During the' decision proce.ss on any major defense 
acquisition, the primary focus is on the development and 
attainment of performance objectives. An accurate acquisi- 
tion cost estimate is an equally significant program 
parameter that must be considered in detail during this 
decision process. Standardized, accurate and detailed cost 
data are indispensible to analysts required to develop 
reliable cost estimates. 

In July 1970, the Defense Blue Ribbon Panel Report 
stated, "the extent of availability of such (cost) data in 
usable form is a limiting factor on the potential accuracy 
of cost predictions." (NAVMAT P-5241, 1973, p. i) The 
Contractor Cost Data Reporting (CCDR) system was established 
by the Office of the Secretary of Defense in 1973 to provide 
for "continual improvement in the ability of the Department 
of Defense (DOD) to develop and use valid cost estimates". 
(NAVMAT P-5241, 1973, p. i) This system is intended to 
provide decision makers with a means by which contract costs 
and other related data can be collected to aid in 
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acquisition management. The reporting requirements are 
designed to collect data on defense weapon systems utilizing 
standard data definitions and reporting structures which 
facilitate integration with other defense management 
systems. To provide a common ground from which to view this 
research, the following excerpts from the Contractor Cost 
Data Reporting (CCDR) System (NAVMAT P-5241), and the 
Military Standard Work Breakdown Structures for Defense 
Materiel Items (MIL-STD-881A) are included in this study. 

2 . Purposes of the CCDR System 

The data collected from the Defense Contractors is 
intended to be used by DOD components in establishing cost 
estimating, programming, budgeting and procurement respon- 
sibilities. This data collection effort is to provide a 
common data base and assist the Department of Defense in the 
following areas: 

a. Preparing estimates in support of the Five Year 
Defense Program 

b. Developing independent government cost estimates in 
support of cost and price analyses and contract 
negotiations 

c. Evaluating contractors' proposals 

d. Responding to requirements for summary information to 
the Secretary of Defense concerning selected 
acquisitions to reflect a comparison of current 
estimates, original plans, and current approved 
program costs and 

e. Preparing cost estimates for major system review by 
the Defense Systems Acquisition Review Council (DSARC) 
at each program decision milestone. (NAVMAT p-5241, 
1973, p.3-1) 
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3 . Reporting Structure 



Reporting requirements are differentiated in 
accordance with the following contract categories: 

a. Category I — Major contracts for Prototypes in Advanced 
Development, Full Scale Development, and Production, 
within programs which are estimated in the Five Year 
Defense Program to require a cumlative financing for 
Research, Development, Test and Evaluation in excess 
of $200 million or a total procurement investment in 
excess of $1 billion 

b. Category II — Contracts for defense materiel items not 
satisfying the Category I criteria but selected by the 
DOD component for cost data reporting because of 
complexity, criticality, future procurement plans and 
contract value. (NAVMAT P-5241, 1973, p. 1-2) 

The data elements which produce the common data base 
are generated by four reports: 



1. 


Cost Data Summary Report 


DD 


Form 


1921 


2. 


Functional Cost-Hour Report 


DD 


Form 


1921-1 


3. 


Progress Curve Report 


DD 


Form 


1921-2 


4. 


Plant-Wide Data Report 


DD 


Form 


1921-3 



These standardized reports are designed to satisfy a wide 
range of weapon system acquisitions. The following excerpts 
from NAVMAT P-5241 provide a basic description of these four 
CCDR system reports which are submitted at varying frequen- 
cies as negotiated in the contract: 

a. DD Form 1921 — Cost Data Summary Report 
(Figure 2-1) 

Primarily designed for Category I contracts, the 
Cost Data Summary Report summarizes all activities included 
in the contract and aggregates cost against the reporting 
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elements selected from the work breakdown structures defined 



in MIL-STD-881 and specified in the contract. WBS elements 
below Level 3 of MIL-STD-881 may be designated for CCDR but 
should be limited to those for which cost data can be 
realistically utilized. The Cost Data Summary Report is 
also used to present the contractor's program estimate for 
RFP ' s , program reviews, or special studies in accordance 
with the fiscal years and quantities specified by the DOD 
component for the total program. (NAVMAT P-5241, 1973, p. 3- 
3) 

b. DD Form 1921-1 — Functional Cost-Hour Report 
(Figure 2-2) 

The Functional Cost-Hour Report is the means of 
identifying and collecting comparable functional costs, 
e.g., engieering, tooling, manufacturing, for (1) specific 
contracts and (2) estimates for the fiscal years and 
quantities specified by the DOD component for the total 
program. Reports may be required for recurring, non-recurr- 
ing, and total costs, as determined and specified by the DOD 
contracting component. (NAVMAT P-5241, 1973, p. 3-9) 

c. DD Form 1921-2 — Progress Curve Report 
(Figure 2-3) 

The Progress Curve Report provides a unit or an 
average unit cost of the unit or lot accepted during the 
report period. All costs reported on this form are recurr- 
ing. (NAVMAT P-5241, 1973, p. 3-17) 
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Figure 2-1. 



DD Form 1921 



Cost Data Summary Report 



18 



CLAISiriCATION 



SCCUKITY CC*S«^lC ATIOI* 0»tB No. 22H037t 













L PROGRAM 














L REPORT AS OF 






FUNCTIONAL COST - HOUR REPORT 






































j CONTRACT 








a 


RFP 










1. OOLLAin IN 


4. hours in 


c 


] PROGRAM ESTIMATE 


















< 

z 




J recurring I , TOTAL 


>. Q RDTAE 




□ 


PROCUREMENT 


^ OTHER 




o 

u 


i. MULTIPI^ rEAN CONTRACT 
n YES ^ HO 


10. QJ PRIME/ ASSOCIATE Q SUBCONTRACTOR 
ZIP 


XL NAME OF CUSTOMERrS.«*eM»r«cw» ••• 






3. FY FUNDED: 
































12. NEPONTING EL£MENT(S) 










ADJUSTMENTS! 
TO PREVIOUS! 
REPORTS 


1 CONTRACTOR 


SUBCONTRACTOR j 

OUTSIDE PROD. A SERVL 


1 TOTAL 




functional CATEGONIES 


TO 

DATE 


AT 

COMPL. 




TO 

DATE 




AT 

COMPL. 




TO 

DATE 


AT 

COMPL. 














b 














• 




f 


9 




ENGINEERING 


1- . 1 










1 




U DIRECT LABOR HOURS 


1 


1 


1 


1 1 


1 i 




2. DIRECT LABOR DOLLARS 


is 1 


IS 


IS 




S 




s 


1 


t 




it 




1. OVERHEAD 


it 


K 1 


It 


It 


i 


s 


i 


IS 


t 




A. MATERIAL 


t. 


t 


IS 


it 


1 


t 


1 


it 


t 




5, OTHER DIRECT CHARGES 


IS I 


s 1 


it 




t 


Is 




s 


IS 




A. TOTAL ENGINEERING DOLLARS 


IS 


i 


s 


1 


Is 


is 


ts 




s 


Is 




TOOLING 




t 


■ i • 


■: , -I 


i \ ■ 


1 




i 




•. • ■ i 






L-:-. 




7. DIRECT LABOR HOURS 








1 




1 




1 




1 




1 




1 


u 




1. DIRECT LABOR DOLLARS 




't 


) 


s 


1 


t 


1 


t 


IS 


1 


s 


1 


IS 




3. OVERHEAD 






s 


IS 


i 


s 


is 


1 


t 


i 


s 


IS 




10. MATERIALS AND PURCHASED TOOLS 


t 


IS 


t 


s 


' 


s 


1 


s 


1 


s 


IS 




IL OTHER DIRECT CHARGES 


s 




s 


IS 




s 


IS 




s 


1 


s 




12. TOTAL TOOUNG DOLLARS 




S 


IS 




s 


r 


s 


IS 




s 




s 












T™T 








1 








. i 




J 




X 1 


VVMWI 1 f 1 












i 












1 




o 

p 


11. direct labor HOURS 1 1 1 1 1 I 


! 

w 1 


1 14. direct labor dollars 


It 


t 


s 


is 


IS 


It 




s 


is 




IS. OVERHEAD 








Is 


If 


It 


IS 


It 


Is 




li. OTHER DIRECT CHARGES 




■ 


s 


IS 


M 


1 


s 




s 




s 




17. total quality CONTROL DOLLARS 


s 


IS 


IS 


S 




s 


IS 




s 




manufacturing 




i 




o 




r 




'■ - i 




t 




- : :■! 






11. DIRECT LABOR HOURS | 


1 1 1 




IS. DIRECT labor dollars 


ts 


1 


s 




s 


IS 


Is 


It 


IS 




20. OVERHEAD 






IS 


! 


s 


IS 


M 


1 


IS 


IS 


1 


t 




2L MATERIALS AND PURCHASED PARTS ! 


It 


1 


t 




t 


It 


it 


It 


r 


t 




22. OTHER DIRECT CHARGES 


IS 


IS 


1 


s 


IS 


s 


1 


s 


It 




21. total manufacturing dollars 


s 


s 


1 


s 


ts 


IS 


IS 




s 












f. 




' 1 




L 












. ■: -l 






24. PURCHASED EQUIPMENT 




It 


It 


!' 


1 


It 




1 


IS 




t 




2S. material overhead 




IS 


It 


1 


s 


IS 




It 




s 


1 


s 




21. other costs not shown ELSEWHERErS««r.^Ml 


It 


IS 


;s 


:s 




1 


It 




27. TOTAL COST LESS GLA 




:t 




1 


M 


1 


s 




.1 




i 


IS 




21. GLA 




IS 


!•' 






•;iS 


ts 




s 


Is 




23. TOTAL COST PLUSGLA 






s 


V 


I.-.' 


1 


is 




IS 




s 


IS 




30. FEE OR PROFIT 






s 


* 




1 


■IS 




s 




1 


1 




s 




3L TOTAL OF UNES 23 A 30 




Is 


* 




“'T 
























_ 




• T 
















DIRECT LABOR MAN'HOURS INCURREO THIS REPORT PERIOD 


► 




EMGlMECflING 

• 


TOOUNG 

k 


QUAUTY CONTROL 
c 


MANUFACTURING 

4 


X 


L TOTAL BEGINNING OF REPORT PERIOD 










- 


2. \ 








< 


^ - - - i 


1 1 


! i 


1 


u 


4. 1 






« 1 
z 


s. 


I 


1 , 


o 

K 


6. total eno of report period 


j 


1 


u 

Ul 




roBm cowriwueo ow neveitse 



DD 1921-1 

Figure 2-2a. DD Form 1921-1 — Functional Cost-Hour Report 



19 



PLANT*V10C LAftOlt AHO OVCRNCAO IHFORMATIOII 











L ]^ECT LABOR 


L PLTVIOE OH 


L DIRECT LABOR 


t PLTVIOE ON 


L DIRECT LABOR 


1 PLTVIOE om 


VOINl 

• 


BASIC 

RATE 

A 


EFF 

RATE 


IMD 

VORK 

4 


RATE 


WORK 

* 


BASIC 

RATE 

t 


EFF 

RATE 

k 


IMD 

VORK 

# 


RATE 

i 


VORK 

f 


BASIC 

RATE, 


EFF 

RATE 


IMD 

VORK 

» 


RATE 

« 


L ENCIHEERINC 
































2. TOOttNG 
































«. OESICN 
































b. fabrication 
































S. QUACITr CONTROL 
































A manufacturing 




i 


I 


























S. MATERIAL 










1 








: 3 












C. C V A 








I 1 








^ 1 














1 


1 1 






1. DIRECT labor j 


2. PLTYmOEOHj 


V DIRECT LABOR j 


|2 RLTWIOEOmI 


1. DIRECT LABOR | 


12 pltwioeoh 




1 EFF 1 
RATEj 


INO 
1 WORK 


RATE 1 


work! 

t i 


BASIC i 

"r'i 


EFF 1 
RATEl 


i 1 

' rate 

1 WORK 1 


WORK 

1 


> basic! 


i EFF i 
RATEj 


INO 

WORK 


RATE 

« 


1 ENGINEERING 


( ! 








1 




i ! 






1 






2. TOOLING 


1 ) 








i 


1 1 


^ i 


1 


i 


• DESIGN 


1 ' 






-■ -i 1 1 


1 ! 


j 1 






• fabrication 


1 






1 1 1 


i 1 








3. quality control 


- -- 1 -1 






- 1 ! 


1 1 1 










4. manufacturing 


1 1 




1 


.1 


1 — 












6 G A A 






i 




1 

1 




-- --i --:i 







NAME or PERSON TO RE CONTACTEO 



Figure 2-2b. 



DD Form 1921-1 
Page 2 



Functional Cost-Hour Report r 



20 



PROGRESS CURVE REPORT 

fKerurrimg Co.tt Only) 



Form Approved 
0MB No. 22R0322 



I 


DOLLARS IN 


IT 


HOURS IN 






s. 


CONTRACT 










L 


REPORT FOR 


MONTHS 


































< 


TOTAL UNITS ACCEPTED PRIOR TO THIS REPORT 




















ENDING: 




































7. 


MULTIPLE YEAR CONTRACT 


9. i PRIME' ASSOCIATE 




u 


SUBCONTRACTOR 




10. NAME OF CUSTOMER 


•*fr> 




LJyes Q no 




7n«w mi»4 Z 


















1. 


FY FUNOEO: 






























IL Ref^RTiNG CLUIEN'nS} 




ITEM 






UNITS/LOTS accepted 


ESTIMATE OF 
NEXT UNIT/LOT 
TO BE 

accepteo 


TO 

complete 

CONTRACT 














1 ■ 








f 


ff 


L MODEL ANO SERIES 




t 


1 






2. FIRST UNIT OF LOT 




i 






1 


1. LAST UNIT OF LOT 




1 


1 


1 




1 




4. CONCURRENT UNITS 




I 




! 




1 






S. 








P" 








1 


1 




CMARACTERimCS L 














1 


i 


1 




7. 














1 


1 1 


i 


contractor data fPER UNIT/LOTI 














■■ . 


s. 


DIRECT QUAUTY CONTROL MAN-HOURS 




1 


1 


I 1 1 


1 ( 


9. 


DIRECT MANUFACTURING MAN-HOURS 








1 i II 


1 i 


10. 


quality control direct labor dollars 


IS 






IS 




Is 1 


Is 




IS j 


1 s 


Is 


lu manufacturing direct labor dollars 


Is 






iS 




is 


i s 




1* - i 


I s 


[ 


S 


la. RAW material a purchased PARTS DOLLARS < 


s 




IS 




IS 


1 s 




IS 1 


s 


1 s 




PURCHASEO EQUIPMENT DOLLARS 


s 






s 




s 


I 




. S 1 


S 1 s 


114. TOTAL DOLLARS 






s 






s 




*s 


IS 




IS is 


1 s 


subcontract/outsioe proo. a serv. i 






•: -i 








t 










IS. direct QUAUTY control MAN'MOURS j 


; , . , 1 


IL direct manufacturing man- 


HOURS 1 


i 1 i i ! i i 


17. 


. TOTAL man-hours 




1 


1 ! i i 1 1 


U. QUAUTY CONTROL DIRECT LABOR DOLLARS i 


s 




1 


s 


IS 


>s 


Is i 


s 


IS 


19. 


manufacturing direct labor dollars 1 


s 




IS 


IS 1 


s 


is IS 


! S 


20. 


RAW MATERIAL A PURCHASEO PARTS DOLLARS ! 


s 




1 


s 


IS 1 


s 


IS Is 




s 


2L PURCHASED EOUIPMENT DOLLARS i 


s 




»s 


IS 


s 




'S 


s 




s 


22. TOTAL DOLLARS 1 


s 




IS 


IS 1 


IS 




i S 1 


i 




s 


UNIT TOTAL □ AVERAGE □ 1 


1 ■ ... .1 .1 1 ~ 


a. DIRECT QUAUTY CONTROL MAN-HOURS 


( 


^ ^ ^ 


24. DIRECT manufacturing MA»»-H0URS 


! 


1 1 


1 


2%. 


total man-hours 




1 


1 ^ ^ ^ 


1 


2A. QUALITY CONTROL DIRECT LABOR DOLLARS 1 


s 




IS 


IS Is 






s 


Is 


27. 


manufacturing DIRECT LABOR dollars |S 




is 




s 1 


s 


S IS 


I 


J 


a. RAW MATERIAL A PURCHASEO PARTS DOLLARS ) 


s 




1 


s 


i 


J 


s 


1* -1 


s 


L 


s 


a. 


PURCHASED EQUIPMENT DOLLARS i 


s 




1 


s 


i 


s 1 


s 


is ' 


s 


is 


u. 


total DOLLARS 




IS 




IS 




S ' 


s 


IS 


s 


i s 


31. 


% subcontract or outside prdd. a serv. 1 






1 




1 








i 




r 






MFG FLOW TIME 


i 
















■ ■•--I 










32. START ( 1 1 ( 1 1 




33. FINISH 1 1 1 1 ' 1 


«e 


34. i ) 1 : 1 


1 1 


O' 


3S. 












i 




o 


3C. 
















o 

31 


37. 


















U. 


















39 

















DO 1921-2 



Figure 2-3a. DD Form 1921-2 — Progress Curve Report 



21 



rm 


UNITS 


/LOTS ACCEPTED 


ESTIMATE OF 
NEXT UNIT/LOT 
TD DE 
ACCEPTED 

1 


TO 

COMPLETE 

contract 

f 


• 


k 


« 


4 


• 


FERFOFMAACE DATA (PER UHfT/LOT) 














40. STAMDARO HOURS 














4L VARIANCE 














SCHEDULE OF RELEilSE DATES 


EN0IHEERIN6 

t 


MATERIAL 

k 


T00UN6 

e 


MANUFACTURIH6 

4 


L PLANNED 










2. ACTUAL 






1 





NAME OF FEffSOM TO «E COfTTACTEO 



ftccunirr cl a&xi ric atiom 

Figure 2-3b. DD Form 1921-2 — Progress Curve Report, 
Page 2 



22 



d. DD Form 1921-3 — Plant Wide Data Report 

(Figure 2-4) 

The report shall be prepared based on the 
contractor's accounting system and the estimating procedure. 
The reporting dates should be established to coincide with 
the contractor's fiscal year. It will be submitted for 
Category I contracts only. This report is a standardized 
plant-wide overhead report which replaces the various types 
of overhead analyses now provided to Administrative Con- 
tracting Officers (ACOs) for major acquisition. (NAVMAT P- 
5241, 1973, p. 3-24) 

The Contractor Cost Data Reporting system 
pamphlet, NAVMAT P-5241, explains how to complete these 
forms and lists definitions of cost data elements to include 
engineering, tooling, quality control and manufacturing. 

4 . Reporting Elements 

The reporting elements required for data collection 
are included in the Request for Proposal and/or the initial 
contract. The level of detail to be included in the 
contractor cost reports will be limited to that which can be 
realistically generated by the contractor and utilized by 
the appropriate DOD contracting component. 

Reporting .elements for Category I contracts are 
selected from the Work Breakdown Structure (WBS) as defined 
in Military Standard — Work Breakdown Structures for Defense 
Materiel Items (MIL-STD 881). Reporting elements for 
Category II contracts are to utilize the WBS elements 
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whenever possible or other criteria most readily reported by 
the contractor using their existing management and account- 
ing systems . 

5 . Direct Labor Hour Definitions 

a. Engineering Direct Labor Hours 

The hours expended in the study/ analysis/ 
design , development , evaluation/ and redesign of the 
specified reporting element. Includes the prepara- 
tion of specifications, drawings, parts lists, 
wiring diagrams, technical coordination between 
engineering and manufacturing, vendor coordination, 
test planning and scheduling, analysis of test 
results, data reduction, and report preparation. 

This also includes the determination and 
specification of requirements for reliability, 
maintainability and quality control. (NAVMAT P- 
5241, 1973, p. 4-1) 

b. Tooling Direct Labor Hours 

The hours expended in the planning, design, 
fabrication, assembly, installation, modification, 
maintenance and rework of all tools, including 
assembly tools, dies, jigs, fixtures, master forms, 
gauges, handling equipment, load bars, work 
platforms (including installation of utilities 
thereon), and test equipment (such as checkers and 
analyzers in support of manufacturing the specified 
reporting element). This entry includes hours 
expended in the determination of tool requirements, 
planning of fabrication and assembly operations, 
maintaining tool records, establishing make-or-buy 
plans and manufacturing plans on components and 
equipment, scheduling and controlling all tool 
orders, and programming and preparation of 
templates and patterns, and form block manufacture. 
(NAVMAT P-5241, 1973, p. 4-2) 

c. Quality Control Direct Labor Hours 

The hours expended in the design and 
implementation of the necessary controls to ensure 
that a manufacturing process produces an item or 
product meeting prescribed standards. Includes 
such tasks as receiving inspection, in-process and 
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final inspection of tools, parts, subassemblies 
and complete assemblies, and reliability testing 
and failure report reviewing; also included are 
such tasks as the establishment of acceptable 
quality level (AQL) and statistical methods for 
determining performance of manufacturing processes. 

The preparation of reports relating to these tasks 
are to be considered quality control effort. 
(NAVMAT P-5241, 1973, p. 4-2) 

d. Manufacturing Direct Labor Hours 

The hours expended on or chargeable to such 
operations as production scheduling and expediting, 
fabrication, processing, subassembly, final 
assembly, reworking, modification, experimental 
production, and installation of parts and 
equipment, power plants, boosters, electronic 
equipment, explosives, and other ordnance items 
(including government furnished equipment) and the 
proving of such equipment, instruments, etc., for 
the specified reporting element. This includes the 
construction of detail parts from raw materials. It 
includes hours expended in the cutting, forming, 
stretching and blanking operations performed on 
material of any kind (metal, wood, plastic, glass, 
cloth, tubing, etc.) to make individual parts. It 
includes bench assemblies of all detail parts, all 
minor and major assemblies, mating or jointing of 
primary sections, installation of special and 
general equipment, instruments and accessories 
performed after the mating and all other 
preparation and/or processing including all 
flashing operations, annealing, heat treating, 
baking, refrigeration, anodizing plating, painting 
and dope operations and preflight and production 
service operations, etc. (NAVMAT P-5241, 1973, p. 

4-3) 

6 . Dollar Values 

The CCDR system provides for the reporting of 
various categories of contractor costs, e.g.. Direct Labor 
Dollars, Overhead, Material, and Other Direct Charges. 
However, because of the fact that all of the data for 
this research would be historical, coupled with the lack of 
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constant dollar data due to inflation, it would be statisti- 
cally more significant to use Direct Labor Hours as the only 
cost data element in this project. 

B. WORK BREAKDOWN STRUCTURE 
1 . Definition 

A Work Breakdown Structure (WBS) is a product- 
oriented family tree composed of hardware, services, and 
data which result from project engineering efforts during 
the development and production of a defense materiel item 
and which completely defines the project/program. A WBS 
displays and defines the product(s) to be developed or 
produced and relates the elements to work to be accomplished 
to each other and to the end product. (MIL-STD-881A, 1975, 
p. 2 ) 

When a weapons system is viewed as a whole, the 
importance of standardized reporting elements is reduced. 
The Level 1 total system hours/cost will remain the same 
regardless of what specific indentured sub-element they are 
reported against. However, as individual reporting elements 
are examined at each level of the Work Breakdown Structure 
the need for these standardized definitions is imperative. 
A lack of standardization will result in inconsistent data, 
and in turn, unreliable cost estimates. 
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2 . 



Purposes of the Work Breakdown Structure MIL-STD 
881A 



As stated in the MIL-STD-881A, the purpose of the 
WBS is to establish criteria governing the preparation and 
employment of work breakdown structures for use during the 
acquisition of designated defense materiel items. These 
work breakdown structures would provide a consistent and 
visible framework that facilitates; 

a. A more effective management and technical base for 
planning and assigning management and technical 
responsibilities by operations within the government 
offices responsible for the acquisition of defense 
materiel items and those contractors furnishing the 
items 

b. More consistent control over and reporting of the 
progress and status of engineering and other 
contractor efforts, resource allocations, cost 
estimates, expenditures, and procurement actions 
throughout the acquisition of defense materiel items 

c. Consideration of total life cycle effects, including 
development, production, activation, operational use, 
and phase-out, when making system development and 
acquisition decisions. 

The uniformity in definition and approach for 
developing the upper three levels of the WBS established by 
this standard is expected to assure compatibility of 
multiple-data requirements. The benefits expected from 
increased uniformity in the generation of work breakdown 
structures and their application to management practices 
will be realized by the improved interpretation and recon- 
ciliation of all reports prepared to this uniform framework 
throughout acquisition of a defense materiel item. (MIL-STD- 
881A, 1975, p. ii) 
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3. Additional Work Breakdown Structure Definitions 



a. Summary Work Breakdown Structure (Summary WBS) 

A Summary Work Breakdown Structure consists of the 
upper three levels of a WBS prescribed by the standard and 
having uniform element terminology, definition, and 
placement in the family-tree structure. The upper three 
levels of a summary WBS have been organized within the 
following categories of defense materiel items: 

1. Aircraft Systems 

2. Electronics Systems 

3. Missile Systems 

4. Ordnance Systems 

5. Ship Systems 

6 . Space Systems 

7. Surface Vehicle Systems. (MIL-STD-881A, 1975, p. 2) 

The three levels specified are defined as 



follows : 

1. Level 1 is the entire defense materiel item; for 
example, the Minuteman ICBM System, the LHA Ship 
System, or the M 109A1 Self-Propelled Howitzer System. 
Usually, Level 1 is directly identified in the DOD 
programming/budget system either as an integral 
program element or as a project within an aggregated 
program element 

2. Level 2 elements are major elements of the defense 
materiel item; for example, a ship, an air vehicle, a 
tracked vehicle, or aggregations of services, (e.g., 
systems test and evaluation); and data 

3. Level 3 elements are those subordinate to Level 2; for 
example, an electric plant, an airframe, the power 
package/drive train, or type of service, (e.g.. 
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development test and evaluation); or item of data 
(e.g., technical publications). 

b. Project Summary Wor)c BreaJcdown Structure 
(Project Summary WBS ) 

Project Summary WBS is a Summary WBS tailored to 
a specific defense materiel item. 

c. Contract Work Breakdown Structure (Contract WBS) 
A Contract WBS is defined as the complete WBS 

for a contract, developed and used by a contractor in 
accordance with this standard and the contract work 
statement . 

d. Project Work Breakdown Structure (Project WBS) 
A Project WBS is defined as the complete WBS for 

the project, containing all WBS elements, related to the 
developments and/or production of the defense materiel item. 

e. Work Breakdown Structure Element 

A work breakdown structure element is a discrete 
portion of a work breakdown structure. A WBS element may be 
either an identifiable item of hardware, set of data, or a 
service- (MIL-STD-881A , 1975, p. 3) 

C. SUMMARY WORK BREAKDOWN STRUCTURE AND DEFINITIONS OF 
AIRCRAFT SYSTEMS 

1 . Project Scope 

As previously discussed, MIL-STD-881A provides seven 
Summary WBS's for use by all contractors and DOD components 
in the development of work breakdown structures for the 
acquisition of defense materiel systems. Since this study 
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addresses only aircraft systems, the WBS in Appendix A of 
MIL-STD-881A will be the only structure presented and 
discussed . 

2 . Summary Work Breakdown Structure 

The following is a Summary Work Breakdown Structure 
for an aircraft system (MIL-STD-881A, 1975, p. 19) with the 
elements of System Test and Evaluation in bold type: 



Level 1 Level 2 Le ve 1 3 

Aircraft system 

Air vehicle 

Airframe 
Propulsion unit 
Other propulsion 
Communications 
Navigation/guidance 
Fire control 
Penetration aids 
Reconnaissance equipment 
Automatic flight control 
Central integrated checkout 
Antisubmarine warfare 
Auxiliary electronics 
equipment 
Armament 

Weapons delivery equipment 
Auxiliary armament/weapons 
delivery equipment 

Training 

Equipment 

Services 

Facilities 

Peculiar support 
equipment 

Organizational/ intermediate 
(Including equipment 
common to depot ) 

Depot (Only) 



32 



Level 1 



Level 2 



Level 3 



System test and 
evaluation 

Development test and 
evaluation 

Operational test and 
evaluation 
Mockups 

Test and evaluation support 
Test facilities 

System/pro ject 
management 

System engineering 
Project management 



Data 



Technical publications 
Engineering data 
Management data 
Support data 
Data depository 



Operational/site 

activation 

Contractor technical 
support 

Site construction 
Site/ship/vehicle 
conversion 

Common support equipment 

Organizational/intermediate 
(Including equipment 
common to depot ) 

Depot (Only) 

Industrial facilities 

Construction/conversion/ 

expansion 

Equipment acquisition or 
modernization 
Maintenance 

Initial spares and 
initial repair parts 

(Specify by allowance list, 
grouping or hardware 
element ) 
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3 . System Test and Evaluation Definitions 



a. System Test and Evaluation 

The System Test and Evaluation element refers to 
the use of prototype, production, or specially fabricated 
hardware to obtain or validate engineering data on the 
performance of the aircraft system. This element includes 
the detailed planning, conduct, support, data reduction and 
reports from such testing, and all hardware items which are 
consumed or planned to be consumed in the conduct of such 
testing. It also includes all effort associated with the 
design and production of models, specimens, fixtures, and 
instrumentation in support of the test program. Test 
articles which are complete units (i.e., functionally 
configured as required by the aircraft equipment) are 
excluded. Development component acceptance, etc., testing 
which can be specifically associated with the hardware 
element, unless these tests are of special contractual or 
engineering significance (e.g., associate contractor), are 
also excluded. (MIL-STD-881A, 1975, p. 24) 

b. Development Test and Evaluation 

The Development Test and Evaluation (DT&E) 
element refers to that test and evaluation conducted to: 

1. Demonstrate that the engineering design and develop- 
ment process is complete 

2. Demonstrate that the design risks have been minimized 

3. Demonstrate that the system will meet specifications 
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4. Estimate the system's military utility when introduced 

5. Determine whether the engineering design is support- 
able (practical, maintainable, safe, etc.), for 
operational use 

6. Provide test data with which to examine and evaluate 
tradeoffs against specification requirements, life 
cycle cost, and schedule. 

DT&E is planned, conducted, and monitored by the 
developing agency of the DOD component. It includes, for 
example, such models and tests as wind tunnel, static, drop, 
and fatigue; integration ground tests, engine military 
qualification tests (MQT), preliminary flight rating tests 
(PFRT), test bed aircraft and associated support; 
development flight test, test instrumentation, test 
equipment (including its support equipment), chase aircraft 
and support thereto, etc. (MIL-STD-881A, 1975, p. 24) 
c. Operational Test and Evaluation 

The Operational Test ' and Evaluation element 
refers to that test and evaluation conducted by agencies 
other than the developing command to assess the prospective 
systems' military utility, operational effectiveness, 
operational suitability, logistics supportability (including 
compatibility, interoperability, reliability, maintainabili- 
ty, logistic requirements, etc.), cost of ownership, and 
need for any modifications. Initial Operational Test and 
Evaluation (lOT&E) conducted during the development of a 
weapon system will be included in this element. This 
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element encompasses such tests as flight tests, sea trials, 
etc., and support thereto, required to prove the operational 
capability of the deliverable system. It also includes 
contractor support (e.g., technical assistance, maintenance, 
labor, material, etc.) consumed during this phase of 
testing. (MIL-STD-881A, 1975, p. 24) 

d. Moc)cups 

The Mockups element refers to the design 
engineering and production of system or subsystem mockups 
which have special contractual or engineering significance, 
or which are not required solely for the conduct of one of 
the above elements of testing- (MIL-STD-881A, 1975, p. 24) 

e. Test and Evaluation Support 

The Test and Evaluation Support element refers 
to all support elements necessary to operate and maintain 
systems and subsystems during flight test and evaluation 
which are not consumed during the flight-testing phase and 
other support requirements that are not allocable to a 
specific phase of testing. This element includes, for 
example, repairable spares, repair of repairables, repair 
parts, contractor technical support, etc., not allocable to 
preceding test and evaluation elements. Operational and 
maintenance personnel, consumables, special fixtures, 
special instrumentation, etc., which are utilized and/ or 
consumed in a single element of testing and which should. 
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therefore, be included under that element of testing are 
excluded. (MIL-STD-881A, 1975, p. 25) 
f. Test Facilities 

The Test Facilities element refers to those 
special test facilities required for performance of the 
various developmental tests necessary to prove the design 
and reliability of the system or subsystem. This element 
includes for example, engine test fixtures, white rooms, 
test chambers, etc. The brick-and-mortar-type facilities 
allocable to industrial facilities are excluded. (MIL-STD- 
881A, 1975, p. 25) 
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III. COST ANALYSIS PROBLEMS IN TEST 
AND EVALUATION OF AIRCRAFT 



A. INTRODUCTION 

The Cost Analysis Division of the Naval Air Systems 
Command (NAVAIR) in the early stage of concept formulation 
initiates action to estimate costs and programmatics. These 
concepts begin prior to the issuance of the initial request 
for proposals when there is a perceived need for a new 
aircraft system to the end of the demonstration/validation 
phase of the aircraft system development life cycle. 

The current practices in data collection and standar- 
dization have been primarily designed for cost analysis and 
estimation. There are several reasons for this lack of 
analytical capability. First, definitions of costs elements 
are sometimes ambiguous. Too frequently, there is a lack of 
mutual effort to eliminate this ambiguity, which often leads 
to a lack of standardization within contractors ' reports 
required by the Department of Defense (DOD). In addition to 
their ambiguity, MIL-STD-881A definitions apply only to 
Levels 1, 2, and 3 of the Work Breakdown Structure (WBS). 
Below Level 3, definitions are generally tailored to the 
specific contract. This lack of specificity, for whatever 
reason, makes it difficult to normalize data submitted under 
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different contracts, which results in the inability to 
effectively support trade-off analysis in the cost estimat- 
ing process. NAVAIR established a CCDR Committee to enforce 
compliance and provide oversight to ensure consistency 
across programs, where appropriate. Figure 3.1 reproduces 
the current draft generic structure of the WBS used by 
NAVAIR when establishing the Project Summary WBS for use in 
estimating/negotiating individual aircraft programs. The 
final decision as to whether or not this WBS is used in the 
negotiation of a program rests with the Program Manager. 

B. THE COST ANALYSIS PROCESS 

1 . Description of Current Practices 

a. Development of the Project Work Breakdown 

Structure 

The cost analysis process is triggered at the 
beginning of the conceptual phase of the Defense Material 
Acquisition process. A Project Summary Work Breakdown 
structure is developed by the DOD component utilizing the 
category summaries in the appendices of MIL-STD-881A . This 
Project Summary WBS will be identified to the concerned 
contractors during RFP or solicitation. This proposed 
structure will be negotiated with the contractors, who may 
recommend changes to the proposed Contractor WBS that they 
believe would improve its effectiveness in achieving the 
goals of the system acquisition. The negotiated Contract 
WBS will then be extended as far below Level 3 as required 
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Level 

VJBS # 1 2 3 4 5 6 



2000 

2100 

2110 

2111 

2112 

2113 

2114 

2114.1 

2114.2 

2114.3 

2120 

2121 

2122 

2130 

2131 

2132 

2140 

2141 

2142 



Systems Test & Evaluation (ST&E) 

Development Test & Evaluation 
Contractor Flight Tests 
Instrumentation 
Functional Ground Checkout 
Flight Test & Demonstration 
Flight Test Support 
Flight Test Spares & Repair 
Parts 

Support Equipment 
Contractor Technical/ 

Maintenance Services 
Wind Tunnel Article & Test 
Wind Tunnel Article 
Wind Tunnel Test 
Static Article & Test 
Static Article 
Static Test 

Fatigue Article & Test 
Fatigue Article 
Fatigue Test 



Figure 3-lA. AIR-524 Standard Aircraft CCDR Work 

Breakdown Structure Format 
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WBS 

2150 

2151 

2152 
2160 

2170 

2171 

2172 

2173 

2174 

2175 
2180 
2190 
21AO 
21_0 
2200 
2210 
2220 
2230 
2300 
2400 
2410 



2420 

2430 

2500 

2510 

25_0 

2600 



Level 

L 2 ^ 4 5 6 

Drop Article & Test 
Drop Article 
Drop Test 

Simulation Testing 
Avionics Integration Testing 
Test Bench/Laboratory 
Flying Test Bed 
Air Vehicle Equipment 
Avionics Test Program 
Software 

Navy Flight Test Prelim DT II 
Navy Flight Test Final DT II 
Navy Flight Test DT III 
Other DT&E (Specify at Level 4) 
Operational Test & Evaluation 
Preliminary OT II 
Final OT II 
OT III & IV 
Mockups 

Test & Evaluation Support 

Test & Evaluation Spares & Repair 
Parts 

Test & Evaluation Support Equipment 
Test & Evaluation Technical/ 
Maintenance Services 
Test Facilities 

Avionics Integration Facility 
Other (Specify at Level 4) 

Other ST&E (Specify) 

Figure 3-lB. AIR-524 Standard Aircraft CCDR Work 

Breakdown Structure Format 
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to clearly define the extent of the contract. The Contract 
WBS is combined with the Project Summary WBS to form the 
Project WBS. 

b. Purpose of the Project Work Breakdown Structure 

The Project Summary WBS, and its derivatives, 

are established early in the acquisition process to provide 
a managerial and technical framework for all activities 
throughout the acquisition life cycle. DOD Project Managers 
are directed to utilize these work breakdown structures, 
"as a coordinating medium in planning for further systems 
engineering, resource allocation, cost estimates , contract 
actions, and work execution. The reporting of progress, 
performance, and engineering evaluations, as well as 
financial data, shall be based on the Project WBS". (MIL- 
STD-881A, 1975, p. 5) 

c. Initial Cost Estimates 

An initial part of the acquisition process 
involves the Cost Analysis Division of NAVAIR. This 
division is responsible for providing an initial rough order 
of magnitude cost estimation of the proposed system, which 
will be continually refined toward absolute accuracy 
throughout the program life cycle. Once a program is 
initiated, the Cost Analysis Division, based on system type 
and acquisition phase, assigns analysts to perform more 
detailed cost estimating and establish the initial cost 
reporting requirements. "During preparation of the Request 
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for Proposals (RFP) or solicitation, the procuring activity, 
should determine the CCDR requirements. This involves 
assessment of cost estimating needs and the contents of the 
data elements which can be generated with the four reporting 
forms". (NAVMAT P-5241, 1973, p. 2-1) The initial task 
reports are further refined by a selected CCDR steering 
committee composed of cost experts. The Program Analysis 
Evaluation (PAE) section of the Office of the Secretary of 
Defense (OSD) reviews, directs modifications if necessary, 
and approves the CCDR. 

d. Reporting Requirements 

With some exceptions, the data from contractors, 
normally down to Level 3, are submitted to the NAVAIR Cost 
Division every six months for dissemination and validation 
for internal use. NAVAIR then systematically distributes 
all the CCDR reports to pertinent parties within DOD (e.g., 
OSD, ASN, PAE). This procedure has been continuously used 
since 1966 and there is no indication that this routine will 
be altered in the foreseeable future due to two main 
reasons. First, the current historical data base is seldom 
implemented below Level 3 of the WBS, OSD probably feels the 
need to maintain the integrity of this data base. Second, 
the Office of Management and Budget (0MB) has established 
guidelines requiring OSD to minimize the reporting require- 
ments placed on the contractors by the CCDR system. This 
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seems to concur with the administration policy to reduce the 
volume of bureaucratic paperwork. 

2 . Implementation Shortcomings of the Present System; A 

Cost Analysis Perspective 

The current system requires contractors to report at 
varied WBS levels, however, from a cost estimating perspec- 
tive reporting requirements continue to inappropriately 
fluctuate from one contract to another. Regardless of the 
depth required in the WBS, contractors claim (despite the 
fact that the CIR/CCDR system reporting requirements have 
been known since the mid 60 's) that it is too difficult to 
convert their accounting systems to accommodate the defini- 
tions given by the CCDR system; an argument with which 
NAVAIR strongly disagrees. To further complicate the 
problem, dual source and sub-contracts are often granted. 
Due to their competitiveness, contractors often cannot, or 
even do not want to cooperate with other contractors. 

For a long time, by analyzing all of the elements of 
the work breakdown structure, discrete answers were genera- 
ted to reply to trade-off questions coming from high-level 
authorities (e.g., ASN, SYSCOM) . However, the required data 
are often not available. When they are available, their 
quality is questionable, making it difficult to provide 
reliable estimates for lower level WBS elements. Usually, 
answers to questions require data below Level 3 of the WBS. 
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It is difficult to obtain periodic (six-month 
period) reporting costs below Level 3 of the WBS without 
encountering resistance from OSD and contractors. The 
latter often claim that it would cost too much to maintain 
such a data base. This argument may be true for some 
contractors, but it is not true industry-wide. Interviews 
performed for this study have indicated that most contrac- 
tors have for a long time maintained data bases with monthly 
data for internal use that potentially contain information 
for detailed and frequent estimation needs. Most of the 
contractors interviewed maintain data at least one level 
below that required by the Project WBS. In fact, some of 
these data bases reflect elements down to Levels 8 and 9. 
It would probably be neither as expensive nor as time- 
consuming, as contractors claim, to provide periodic cost 
reports below Level 3, especially considering the fact that 
CIR/CCDR requirements have existed since 1967, and the 
capabilities of modern day computers to reformat/reaggregate 
contractor's cost accounts into CCDR format. 

C. RECOMMENDED COURSE OF ACTION 

1 . Necessity to Break the WBS to Lower Level 

Regardless of fiscal, political and technical 
considerations, it is critical for the cost analyst to have 
access to accurate data below Level 3 of the WBS for 
meaningful data analysis. In Level 2, System Test and 
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Evaluation WBS and distribution of Direct Contract Labor 



Hours of a typical military fixed-wing aircraft. Of the 
four Level 3 sub-elements. Development Test and Evaluation 
accounts for 93.6% of the contract total System Test and 
Evaluation labor hours. 

The Level 3 element Development Test and Evaluation 
(WBS # 2100) had to be broken down to its Level 4 sub- 
elements to capture its major cost WBS elements. By 
referring to WBS # 2119, it can be found that Contractor 
Flight Tests account for 43.3% of Level 3 Development Test 
and Evaluation, which corresponds to 40.6% of Level 2 System 
Test and Evaluation. The three remaining Level 3 sub- 
elements account for only 6.4% of the total contract labor 
hours. In particular, TECHEVAL costs which are on Level 3 
(WBS # 2200) were minimal, amounting to 0.4% of Level 2 
System Test and Evaluation. 

In the long run, it would be advisable to restruc- 
ture the WBS system by revising the hierarchy of the WBS 
elements. One recommendation for revision would be to 
remove Development Test and Evaluation (WBS # 2100) from 
Level 3 and replace it with selected major WBS elements 
currently located on Level 4. In addition to the obvious 
choice of selecting the Contractor Flight Test element to be 
moved to Level 3, other Level 4 sub-elements contain 
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WBS # 



2000 

2100 

2110 

2111 

2112 

2113 

2114 
2116 

2117 

2118 
2119 

2190 

2191 
2200 
2220 
2400 
2500 



TABLE 3-1. SYSTEM TEST AND EVALUATION DIRECT 
CONTRACT LABOR HOUR DISTRIBUTION 



-Level Percentage of Total 

234 Contract Labor Hours 



SYSTEM T&E 100 



DEV TEST 


93.6 




SYSTEM REQ 




9.4 


WIND TUNNEL 




4.3 


STATIC T&A 




5.8 


FATIGUE TSA 




6.0 


A/V SUB TEST 




8.6 


AVNX INT TEST 




6.6 


ARM/WPN INT 




0.7 


FLT SIM PROG 




1.5 


CTR FLT TEST 




40.6 


MISC 




6 . 6 


DROP & ACL LOADS 




3.5 


TECHEVAL 


0.4 




FLT TEST PRO SPT 




0.4 


MOCKUPS 


4.5 




T&E SUPPORT 


1.5 
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significant percentages of the total System Test and 
Evaluation efforts and should also be considered for a move 
to Level 3 of the WBS. 

The prospective of implementing a revised version of 
the WBS system depends much on the personalities of the con- 
stituencies involved in the process, and the organizations 
created around programs . System Commands should ensure that 
CCDR oversight committees are formed to review current 
procedures, and enforce proper implementation. The coopera- 
tion of contractors, as well as DOD components, will be a 
vital part of this effort, which will involve optimizing and 
sharing by all concerned. 

2 . Necessity to Provide Time-Phased Data Reporting 

Data should be reported in more depth, systematical- 
ly, and in a time-phased fashion. Time is a big cost 
driver in some elements. For example. Contractor Test 
Flight costs seem to relate to the length of the test 
periods. A side benefit of this would be that cost esti- 
mators would have an improved historical base for profiling 
cost estimates into annual budget increments to predict 
Research and Development costs for each year. While it is 
difficult to derive an optimal frequency of time series data 
reporting, it would make sense to argue that yearly reports 
are minimally adequate. Monthly data would be preferable 
since they would capture the detailed trend of labor hours 
incurred throughout the system development life-cycle, but 
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might not be cost effective. It is a question of the effort 
required to convert data from contractor to the CCDR system 
format, and the normalization of data across various 
contractor inputs. As a compromise, semi-annual data would 
be acceptable and more cost effective, since all contractors 
have computerized accounting systems to maintain their costs 
internally. Due to the long process of aircraft system 
development, semi-annual data are expected to provide 
sufficiently detailed information for trend analysis. 

3 . Implementation of a well-defined CCDR Data base 
System 

As it is defined in Chapter II, the purpose of the 
CCDR system is to provide the primary data base for use in 
most cost estimating efforts including procurement manage- 
ment activities. As discussed earlier, the realization of 
this effort has been dampened by the lack of standardized 
data format among similar programs below Level 3 of the WBS. 
It seems thus evident that one of the first steps to be 
taken is to elaborate a comprehensive, consistent, and 
precise WBS Elements Dictionary that can be: 

1. Agreed upon by all DOD components as well as the 
Defense Industry 

2. Implemented for cost analysis purposes. 

Ideally, the new definitions would serve as foundations for 
building a data base system that compiles all data in an 
accessible, dynamic, and evolving data base, readily avai- 
lable for retrieval and modification for modelling and 
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analysis. Historically, required reporting elements have 
been inconsistently determined across contracts for similar 
weapon systems. The required levels of reporting within the 
WBS have often been inconsistent among the different DOD 
components. Individual contracts have also been negotiated 
to allow some obscuring of information due to subcontractor 
reporting variances and deficiencies. Labor cost com- 
parisons in terms of dollar expenditures do not provide a 
basis for consistent comparison due to changing economic 
price levels. Variances between contractor methods of 
accounting also generate ambiguity. For instance, some 
contractors may allocate labor hour expenditures for direct 
maintenance and operational support of the test vehicle to 
engineering costs, while others may allocate the same 
function to manufacturing costs. These factors do not imply 
inaccuracy of the data, but do induce inconsistency when 
comparing and correlating individual models of aircraft and 
Defense Contractors. 

The utilization of the Contractor Cost Data 
Reporting system data base to provide explicit estimates for 
sub-elements of aircraft system test and evaluation has not 
been expedient. Lower levels of data than previously found 
in the Work Breakdown Structure should be analyzed in order 
to effectively obtain the desired information and accurate results. 
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IV. RESEARCH METHODOLOGY 



A. OBJECTIVES AND SCOPE OF RESEARCH 

1 . Objectives 

This research seeks to; (i) assess current problems 
in Cost Analysis in Aircraft System Test and Evaluation, 
(ii) establish guidelines for estimating data, (iii) design 
and implement a database structure for Aircraft System Test 
and Evaluation, and (iv) develop parametric cost estimating 
relationships for selected System Test and Evaluation 
elements and four functional cost categories. Specifically, 
this research will focus on; 

1. Defining a sample of pertinent aircraft programs 

2. Formulating a WBS structure/functional cost element 
matrix 

3. Acquiring historical cost data 

4. Organizing the data for analysis 

5. Employing statistical methods to develop cost estimat- 
ing relationships 

6. Documenting data, sources, rationale, and methodology. 

2 . Scope 

This research attempts to apply econometric theory 
and cost analysis in the area of Aircraft Systems Test and 
Evaluation (ASTE) . Multiple linear and logarithmic regres- 
sion are probably major analytical tools. Since data are 
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not readily available from a single source to the detail 
required for econometric studies, it is necessary to 
establish an operational data structure that can be used as 
a basis for this research and facilitate its data collec- 
tion. 

This research also applies the concepts defined in 
system analysis and design to implement decision support 
software for ASTE. Such an implementation would greatly 
help cost analysts interact with the complex and diverse 
data base. Consequently, the reliability of their cost 
estimation would be enhanced, or, at least, limitations of 
estimating methods clearly understood. 

Interviews with defense contractors, Patuxent River 
Naval Air Test Center, Edwards Air Force Base Flight Test 
Center, and on-site data collection are envisaged for the 
first part of the research. Utilization of statistical 
software packages to identify elasticities of cost drivers 
will constitute the second part of the research. 

B . RESEARCH METHODOLOGY 

As Stated in the previous section, this research 
encompasses both analytical and empirical studies. The 
methodology adopted for this research follows a three-step 
process shown in Figure 4-1. The activities of each of the 
three steps are briefly described below. 
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1 . 



Exploration Phase: This phase includes the initial 

statement of the problem, the assessment of its 
importance and feasibility, and the establishment of 
contacts with access to relevant data. 

2. Analysis Phase: This is an analytical approach to 
problem solving that requires the clarification of 
goals, definition of objectives and identification of 
systematic research activities to achieve these goals, 

3. Refinement; This constitutes an iterative process to 
gradually readjust actions required to better meet 
stated objectives. Particularly, this process would 
highlight the sensitivity of decisions, the values of 
the key cost drivers and assumptions on which es- 
timates are base including economic, technical, 
operational, schedule and other problematic con- 
siderations . 




Figure 4-1. Phases of Research Methodology 



C. EXPLORATION 

The Exploration Phase was generated by an attempt to 
define the scope and objectives centering on the issues of 
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controlling and estimating test and evaluation costs. 
However, this research team's focus was not clarified until 
initial meetings with the Cost Analysis Division of NAVAIR, 
held on August 7-8, 1986. Although there is a need to 
develop cost estimators for all of the Systems Test and 
Evaluation elements, it was confirmed at that time that 
initial efforts should be confined to the Contractor Flight 
Test element, and if feasible, expand the research to 
include the Wind Tunnel Article and Test, Static Article and 
Test, and Fatigue Article and Test elements. On-site visits 
at the Naval Air Test Center, as well as interaction with a 
selected group of representatives, demonstrated some 
feasibility of data collection and identified potential cost 
drivers, particularly in the area of aircraft weight and 
speed, and flight test hours. The initial Problem Defini- 
tion in Figure 4-2 was developed following these meetings. 

D. ANALYSIS 

Based on the initial problem definition (Figure 4-2), 
a study was conducted to investigate the economic, political 
and technical feasibility of attaining the defined objec- 
tives. While no insurmountable problems in the financial 
and technical aspects were envisaged, the accessibility of 
contractors' proprietary data was identified as a potential- 
ly insurmountable political consideration that could 
severely constrain the realization of the intended research. 
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PROBLEM DEFINITION 



STATEMENT OP SCOPE AND OBJECTIVES: August 7-8, 1986 

BETWEEN: Naval Postgraduate School 

AND: Naval Air Systems Command 



PROJECT: Cost Analysis for Aircraft System Test and 

Evaluation - Data Collection. 



PROBLEM: There is an increasing requirement from high 

levels that the Navy's aircraft cost es- 
timators/analysts provide explicit estimates 
for the sub-elements of aircraft systems test 
and evaluation efforts. These data are not 
presently available at the levels required by 
the Naval Air Systems Command (NAVAIR). 

OBJECTIVES: Collection of data to facilitate NAVAIR' s 

development of cost estimating relationships 
for the following test and evaluation elements: 
Contractor Flight Test, Wind Tunnel Article & 
Test, Static Article & Test and Fatigue Article 
& Test. 



SCOPE : Project to be completed within six 

calendar months at a cost of no more than 
$45,000. Availability and willingness of 
contractors to provide data that could be 
considered proprietary. Inflexibility of 
NPS Officers' schedules due to curriculum 
requirements . 

PRELIMINARY SOLUTION: 

Visits to Defense Contractors' facilities 
to discuss the problem and to ascertain 
the accessability and availability of the 
required data. Establish a database to 
facilitate the analysis of the data. 



FEASIBILITY STUDY: 

A feasibility study should be conducted 
with the results submitted within two 
calendar weeks. The cost of the feasibil- 
ity study is included in the project scope 
and will not exceed $2000. 



Figure 4-2. Initial Problem Definition 
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To minimize the impact of this political constraint, a 
strategy was formulated to ensure the viability of the data 
collection process by: 

1. Identifying contacts and soliciting their cooperation 

2. Conducting on-site interviews. 

1 . Identifying Contacts and Soliciting Cooperation 

NAVAIR provided a list of twenty-one contacts 
throughout the defense industry. Through phone conversa- 
tions and correspondence (Appendix A) , this list was refined 
and expanded to those that indicated not only an interest in 
the results of the research, but a cooperative attitude 
concerning the request for company data (Appendix B) dealing 
with System Test and Evaluation. 

In general, respondents were highly interested in 
the nature of the project and indicated their willingness to 
assist in data collection and critical evaluation of the 
present cost analysis process. Those who declined to 
participate cited the following reasons; 

1. The lack of available data 

2. The current political climate within the industry, and 

3. On-going contract negotiations. 

The final list consisted of thirteen companies, which made 
available personnel from a wide range of aircraft-related 
disciplines as depicted in Figure 4-3. 
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Job Description 


Number of Contacts 


Cost Estimation 


18 


Data Managers 


7 


Engineers 


11 



Figure 4-3. Interviewees Job Description Breakdown- 



2 . On-site Interviews 

Interviews were conducted at thirteen locations 
throughout the United States. Interview teams consisted of 
at least two research members per visit. Surveying time 
lasted, on the average, approximately three hours for each 
location, varying from forty-five minutes to ten hours. 

To prepare for the interviews, the research team 
brainstormed to establish the following initial list of 
issues that appeared to be important for the research 
methodology. 

1. What are the cost drivers in ASTE? 

2. How do the characteristics and complexity of the 
aircraft relate to the average engineering hour? 

3. What statistical and econometric models can be 
appropriately applied to ASTE? Is multiple regression 
appropriate to this problem? 
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4. Are the military work-breakdown structures pertinent 
to ASTE? 

5. How can a computer system (hardware and software) be 
designed and implemented to conduct testing of the 
models? 

6. How have data been defined and collected in different 
test sites (military and civilian)? Can they be 
filtered, and standardized? What extrapolation method 
should be used for missing data? 

7. How do these cost drivers affect the average engineer- 
ing hour per hour of flight? 

8. Is there any correlation between the number of 
instrumentation channels and the number of engineering 
hours? 

9. Is there a linear cost relationship between different 
types of aircraft with respect to their complexity and 
characteristics? 

10. How sensible and stable are the elasticities of the 
cost drivers? 

11. What are the implications of the findings? 

12. Where is it most cost-effective to conduct future 
military ASTE — at the contractors facilities or at the 
established flight test centers. Is the answer the 
same for both flight test centers? 

13. Do the findings have an impact on the service's opera- 
tional test and evaluation? 

14. How does inflation affect the predictive power of the 
findings? 

This initial list of questions was sent to interested 
parties. Additionally, to improve our own understanding of 
industry practices and procedures, the list of issues were 
distilled to the following set of questions that could be 
addressed in on-site interviews. 

1. Would you provide general specifications of all your 
company's aircraft to include; aircraft type and mis- 
sion, thrust to weight ratio, speed, size, ceiling. 
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combat radius, conunonality/GFE between models, and use 
of exotics and/or composites? 

2. What methodologies do you use for accounting and 
record keeping of cost items (development and test- 
ing)? 

3. How are these accounting and record keeping items 
defined? Are these definitions standardised company 
wide? Do you have a computerized cost accounting 
system? 

4. What methodologies do you use to extrapolate cost 
estimators from these data? 

5. In the area of avionics, what impact do variations in 
the following items have on the labor hours and flight 
test hours of your aircraft systems ; 

a. The number of instrumentation channels. 

b. Number of lines of delivered software code. 

c. The number of separate CPU's installed in the 
aircraft. 

d. New electronic technology (e.g., VSLI ) . 

e. What hardware integration factors affect the 
cost (i.e., is a decentralized system more cost- 
effective than a centralized system?). 

f. Power usage. 

6. What do you believe are the most important cost- 
drivers? Could you prioritize them? 

7. If due to cost constraints you had to reduce test and 
evaluation activities, what would be your primary 
candidates for elimination? Which activities do you 
consider indispensible to test and evaluation? 

8. What would you do to reduce costs? 

This set of questions permitted a cross comparison among 
companies. More interesting, it was found that, as the 
interview progressed, the domain knowledge of the research 
team expanded exponentially which improved the quality of 
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the remaining discussions. The contacts below also raised 
numerous issues they thought significant enough to included 
in the research. 



12 November 


1986 Rockwell International Corporation 

Los Angeles, California 


12 November 


1986 Boeing 

Seattle, Washington 


13 November 


1986 Air Force Flight Test Center 

Edwards Air Force Base, California 


13 November 


1986 McDonnell Douglas Corporation 

Long Beach, California 


14 November 


1986 Lockheed California Company 

Burbank, California 


17 November 


1986 Rockwell International 

Columbus, Ohio 


17 November 


1986 General Dynamics Corporation 

Fort Worth, Texas 


17 November 


1986 LTV Aerospace and Defense 

Dallas, Texas 


18 November 


1986 Grummann Aerospace Corporation 

Bethpage, New York 


18 November 


1896 Fairchild Aircraft 

Farmingdale, New York 


18 November 


1986 McDonnell Aircraft Company 

St. Louis, Missouri 


19 November 


1986 Lockheed Georgia Company 

Marietta, Georgia 


20 November 


1986 Naval Air Test Flight Center 

Patuxent River, Maryland 



As the interview process continued, important issues 
converged demonstrating a industry-wide commonality of 
problems (see Chapter V). 
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E. REFINEMENTS 



A post-interview meeting was convened at NAVAIR to 
assess the results of the contractor visits. The research 
team reported to the sponsor the nature and amount of data 
that were, and could be, collected. Also, the sponsor was 
briefed on the areas considered most important by contrac- 
tors ( see Chapter V), which they had recommended be added 
to the present cost process to make it more reliable and 
accurate, in particular: 

1. Software 

2. Avionics 

3. Instrumentation channels 

4. Requirements to meet performance guaranties 

5. Contract warranties required by the Government 

6. Fixed-priced contracts and 

7. Combined DOD / Contractor Flight Testing. 

It became obvious that these new dimensions could 
contribute to the rapid growth of the present project scope. 
If they were all included, it would no longer be feasible to 
accomplish the totality of the project given the constraints 
imposed. Therefore, the team requested the sponsor to 
refocus the problem and set research priorities in light of 
these new factors. It was then stated that the focus would 
remain on Contractor Flight Test, with Wind Tunnel Test, 
Static Test, and Fatigue Test studies as secondary goals. 
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In order to achieve this restated goal, interpreta- 
tion, review, and standardization of data became indispen- 
sible. Due to his expertise, the sponsor offered his 
assistance. This operation is time intensive, requiring 
tedious and systematic efforts. In the meantime, a follow- 
up letter was sent to contractors seeking time-phased data 
(Appendix C). In parallel, a data base was designed to 
accommodate the data received from the contractors (see 
Volume II, Chapter III). 
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V. COST DRIVERS IN SYSTEM TEST AND EVALUATION 



A. INTRODUCTION 

Past studies have primarily considered, weight, speed 
and the number of aircraft as the most statistically 
significant cost estimators. In fact, other cost drivers — 
such as avionics, software, management strategies--have 
recently emerged as important, if not more important than 
these three drivers. Unfortunately, due to the current 
practices of data collection, it is impossible to statically 
quantify the new cost drivers. The only approach that 
remains appropriate to evaluate the importance of these 
drivers is to use the Delphi technique to solicit expert 
opinions in the aircraft system cost estimation area. 
Interviews with aerospace engineers, pilots, Test and 
Evaluation managers from Defense Contractors and DOD, test 
flight centers, and defense contractors have been conducted 
to brainstorm issues regarding Aircraft Systems Test and 
Evaluation costs. 

B. IDENTIFICATION OF THE MOST IMPORTANT COST DRIVERS 

1 , Mission as the Determinant of Cost Drivers 

Mission is the cornerstone upon which any major 
program is built. Until the mission of the proposed 
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aircraft system is defined, it is not possible to determine 
specifications; even at a broad level. However, once the 
mission is known, then the many factors that form the 
complex world of Defense Systems Development and Acquisition 
begin to come together. Many of these factors are clearly 
defined and are measurable cost drivers that will be 
discussed later in this chapter. It is important to note, 
however, that there exists an environment within which DOD 
and Defense Contractors operate that effects all programs as 
surely as the more obvious cost drivers. 

One of the major managerial concerns is whether a 
bid can be submitted in such a way that it will not only 
allow the company to develop and produce the system within 
the defined financial constraints, but insure that there is 
a means available to recoup the cost of DOD's "inevitable" 
modifications to the specifications and/or performance 
guarantees. It is no longer a matter of "if" there will be 
changes, but rather a question of "when and how many". The 
Cost Plus Contract provided the contractor with an insurance 
clause, which often proved to be very expensive from the 
government's perspective. The Fixed Price Contract now 
mandated by DOD has removed this insurance clause, provided 
DOD with protection from cost overruns, and has moved the 
"risk factor" to the contractor. Since it should not be 
considered unreasonable for either DOD or the contractor to 
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expect some changes over the long developmental life cycle 
of a major weapon system, the present relationship is one 
punctuated with compromise and conciliation. If it were 
otherwise, the development and attainment of a program's 
performance objectives would be impossible. 

The requirement to provide warranties on an aircraft 
system that has yet to be built, and whose operational life 
is unknown, is another major management concern in the early 
stages of a program. How does an analyst quantify costs, 
for inclusion in a bid, that are so unpredictable? And how 
does management calibrate the experience and expertise of 
its engineers in cost estimators? Are the cost estimates as 
accurate as the specifications and performance guarantees? 

Sub-contractors and secondary vendors are often 
trapped into making guarantees too early in the process. 
Their desire to be a part of the program leaves them with no 
other alternative, and as a result the integrated overall 
mission is typically not met. 

When a ceiling dollar value is included in a Request 
for Proposals there are few contractors who will refrain 
from bidding the indicated ceiling, even if their own cost 
estimators believe that it will be impossible to stay below 
that figure. "If we get our foot in the door, we'll 
renegotiate later." 
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2 . Aircraft Weight 



All companies interviewed indicated that weight 
still constitutes a major cost driver in system development. 
Because of the availability of the data on weights, it has 
been relatively easy to use weight as a cost estimator. 
Dollars and hours per pound have often been used as es- 
timators. Everything else being equal, the larger the 
aircraft the greater the cost, i.e., a bomber, which is 
approximately five times heavier than a fighter, has overall 
costs (Level 2 of the WBS) that are roughly five times more 
than that of the fighter. This interpretation corresponds 
to earlier studies performed by PRC (1967), RAND (1972, 
1975, 1976), and Noah (1973). These studies all ranked 
weight as the primary cost driver. As weight increases, 
cost increases too. However, as weight increases, at some 
point, cost per pound decreases. Additionally, aircraft 
density effects the cost per pound of aircraft. As an 
aircraft is loaded with more sub-systems (e.g.. Avionics), 
cost per pound of the aircraft increases. 

Recent use of composites has altered the statistical 
quality of weight as a cost estimator since historical data 
are associated with metal. It has been hypothesized that as 
more experience is gained with composites, weight will again 
become a more stable and predictable factor because of the 
nature of building with composites. Although testing an 
aircraft component built entirely with composites should be 
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easier, one of the major uncertainties when dealing with 
composites in the Contractor Flight Test phase occurs when 
an test-site modification or repair to an aircraft is 
required. With the increased use of composites in the 
future, weight is expected to move from labor-intensive to a 
automation(capital)-intensive factor. While it is still 
difficult to predict at the present time how the use of 
composites will drive the cost associated with weight, it is 
plausible that developmental testing will progressively 
become less expensive. It is also hard to relate weight of 
composites to cost since there are more parts in non- 
composite aircraft than in composite aircraft. 

3 . Aircraft Speed 

After weight, speed is also recognized as an 
important cost drivers. However, it was identified only by 
70% of the opinions solicited. Thrust being constant, 
weight is related to speed. One of the reasons that speed 
represents an important cost driver is the fact that due to 
the requirement to meet performance guaranties, development 
costs increase at a spiraling rate as engineers work with 
the aircraft at critical speeds. The reduced total weight 
of an aircraft due to the use of composites will decrease 
the thrust required to attain the same speed of a non- 
composite aircraft, and as a consequence, developmental cost 
in this area should be diminished. 
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4 . Avionics Complexity 



With one exception, avionics complexity was another 
cost driver that emerged from the interviews. Advanced 
electronic technology has resulted in improved reliability 
of avionics and substantial amelioration in troubleshooting 
and maintenance procedures. Test effectiveness, test 
vehicle flight frequency, and data validity have greatly 
improved. Many contractors claimed that achievement in 
avionics have been responsible for their recent gain in 
program planning effectiveness and sortie rates of their 
flight test programs. 

While software-intensive avionics substantially 
innovates the capabilities of aircraft systems, it also 
presents enormous testing requirements. For example, a high 
percentage of test work accomplished by the Air Force for 
the F-15, F-16, and B-1 is related to software. When the 
volume allocated to avionics within the airframe is con- 
stant, advanced technology decreases the weight and in- 
creases the avionics capabilities. This implies that while 
more complex aircraft can be built without affecting the 
costs related to increased weight and speed, the complexity 
of avionics will drive the total system costs up. Avionics 
is a difficult cost to estimate due to the difficulty in 
estimating software. 
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5. Software 



Indeed, an important factor that has emerged from 
the increased avionics complexity is software. Unlike 
weight, there is a predominant concern on developing the 
ability to accurately estimate software costs. This element 
is regarded as the most difficult cost driver to estimate. 
First, there is little consistency among companies in the 
way aircraft system software is developed. Some develop by 
total system concepts, others modularize by functional areas 
(e.g., navigation, fire controls, weapons), and still others 
concentrate on implementing interfaces between blackboxes 
developed by various sub-contractors. Second, estimation 
methods range from dollar s-per-word to parametric models 
based on total lines of codes. These models include COCOMO, 
SLIM, RCA/PRICE/S, IBM Walston-Fe lix , and - Boeing Computer 
Service. (Wolverton, 1980) However, the complete lack of 
historical data and experience with Aircraft System Test and 
Evaluation software makes it virtually impossible to apply 
these parametric models. (For a survey of parametric models, 
see Volume II, Chapter II.) The estimation problem is 
further complicated by the fact that software costs depend 
on the number of modules, the size of the modules, the 
degree of integration complexity, and the number of proces- 
sors in distributed systems available to run the integrated 
software system. 
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It is felt that the mission determines the software 
costs. For example, a training mission does not require a 
complex software structure as compared to an all-weather 
attack mission. Consequently, software costs for the 
trainer are much less expensive than for those of the all- 
weather attack aircraft. In addition to the primary 
mission, the specific number of mission tasks the aircraft 
is capable of performing does play an important factor. For 
example, in general, the ECM mission would derive a higher 
software cost than the ASW mission. However, the software 
cost of the AWACS is higher than the E-2C. These two 
aircraft perform similar missions but on different scale. 
Another comparison would be between the F/A-18 and the A-6. 
If both aircraft were built from the ground up using today's 
technology, the software cost of. the multi-mission F/A-18 
would probably be substantially higher than the A-6. 

A strategy to deal the increasing cost of testing 
software intensive system is to use simulation. Research 
conducted at the Air Force Flight Test Center, Edwards Air 
Force Base argues that: 

1. Three-fourths of all software problems are resolvable 
on the ground at a fraction of a cost of a flight 

2. A reduction in test flying hours translates into a 
reduction of test costs and an acceleration of the 
test schedule 

3. The costly and inefficient fly-fix-fly approach is 
minimized 
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4. Ground testing is more efficient than flight testing 
because the experiment is controlled, repetition of 
test conditions is rapid and simple 

5. Flight test time is used more effectively by isolat- 
ing/keying on risks areas and smarter profile plan- 
ning. (Adolf and Montgomery, 1985, p.3) 

Tables 5-1 and 5-2 show expected cost savings for a typical 
fighter test program utilizing simulation of software- 
intensive systems. 

A second approach to resolving the problem of 
software cost estimation would be for the entire Defense 
Industry to officially endorse the utilization of structured 
analysis and design methodology through functional decom- 
position. (Yourdon, 1986) Such decomposition by functions 
would make it easier for software developers to capture the 
complexity of each function and systematically integrate 
them as a total system for cost estimation. Similar 
software engineering techniques and experience will result 
in more standardized data collection needed for future 
estimation. 

Another potential area of improvement is the 
implementation of distributed systems using parallel 
processors. While no statistical evidence was gathered to 
assess the impact of separate CPUs on aircraft testing, the 
majority of interviewees believe that the conduct of tests 
and the probability of completion during a specific flight 
were enhanced when several CPUs were installed, permitting 
the successful testing of some aircraft systems when others 
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TABLE 5-1 



aircraft 


TEST FLIGHT 
FREQUENCY 
(FLIGHTS/MONTH) 


TEST AIRCRAFT 
OPERATIONS COST 
($lOOO/HOUR) 


?LICHT TEST FIXED 
SUPPORT COST 
($1000 /MONTH) 


PROCRA^OtABLE 
CAPACITY 
(1000 WORDS) 


CURRENT 

generation 

FIGHTER 


10 


20 


SI »000 


300 - 700 


CURRENT 

GENERATION 

aOMBER 


5 


100 


$2,000 


600 - 800 


NEXT 

GENERATION 

FIGHTER 


10 


40 


$1,500 


1,000 



TABLE 5-2 



COST OF FLIGHTS; 




(250 FLIGHTS) (1,25 HRS/FLT) ($20. 000/HR) - 


$ 6.250,000 


LESS COST OF SIMULATION; 




(250 FLIGHTS) (1 HR/FLT) ($4, 000/HR) 


$ 1,000,000 


NET SAVINGS FROM REDUCED FLIGHTS: 


$ 5,250,000 


(20 AIRCRAFT WNTHS) C S 1 ,000 ,000 /MONTH) 


$20,000,000 


TOTAL SAVINGS; $5,250,000 ♦ $20,000,000 


$25,250,000 
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have failed. In the past, a central CPU failure precluded 
continuation of the testing process. 

6 . Power Supplies 

Power supplies play a vital role in the operation of 
all aircraft. These power sources provide a range of 
outputs depending on the requirements and number of the 
devices they support. This accounts for the varied capabil- 
ities, sizes and quantities of power supplies that can be 
found in any aircraft system. 

The majority of the volume and weight of a typical 
power unit is taken up by the filters required to ensure 
that delivered current is within specified operating ranges. 
When compared to the technological advances made in other 
avionics areas, it is obvious that there has been very 
little progress in the area of reducing or eliminating these 
unit filters, thereby improving the overall efficiency, 
presently necessary to meet the requirement specifications. 
If more efficient power units were developed, the number of 
these sources could be reduced, thereby reducing the total 
weight and volume of avionics in the aircraft. Also, if 
power sources were more stable and cleaner, testing costs 
would be reduced. 

7 . Data Reduction 

Data reduction, which is the process of reviewing 
and analyzing the data gathered during testing, presents 
another area of concern. Compounding this problem is^ the 
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increased number of instrumentation channels. With the 
advent of a central bus within the airframe, engineers have 
recognized the potential for monitoring additional data 
points that were not previously feasible. Convinced that 
these data could now be gathered relatively easily, they 
requested that procedures for monitoring additional data 
points be included in the test plan. Most contractors felt 
that this process would be cost effective since it would 
provide a wide cross section of data on each and every 
flight which would allow maximum concurrent testing regard- 
less of the primary purpose of that flight. 

However, the current technology that assists in 
analyzing data has not progressed as fast as technology's 
ability to produce raw data. Collected data have become so 
voluminous and overwhelming to the extent that it has become 
impossible for the engineers to properly analyze all of the 
data unless the time constraints of the test program were 
extended . 

8 . The Number of Test Aircraft 

The number of test aircraft also constitutes an 
important cost element. Historically, it was observed that 
this cost driver was considered one of the most significant 
in flight test cost estimation, along with weight and speed. 
(PRC, 1967; RAND, 1972; RAND 1975) As the number of 
aircraft increases, the number of separate test flight 
augments and drives the total cost higher. Aircraft are 
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instrumented to perform specific portions of the test plan 
and therefore cross utilization is infeasible unless 
aircraft are modified for other specific tests. The 
determination of an optimal number of aircraft is extremely 
difficult since the cost relationship is not linear. If the 
number of aircraft is below the testing capacity then 
testing resources are wasted. Conversely, when the number 
of aircraft exceeds the testing capacity, there is a loss of 
the maximum utilization of test aircraft assets and ex- 
perienced personnel. As an attempt to reduce costs, 
contractors argue that the number of aircraft used in flight 
testing should be held to the minimum required to meet the 
test plan schedule. They felt that DOD as a whole should not 
have as much input in determining the number of test 
aircraft as it is presently exercised. 

9 . Delivery Schedule 

It is expected that this cost element is gaining in 
importance due to the: 

1. Increasing complexity of aircraft 

2. Requirements to meet all performance guaranties imposed 
by the Government prior to the first delivery of pro- 
duction aircraft. 

As a proposition to reduce this problem, it was proposed by 
several contractors that aircraft be delivered in a block 
(phase) program. A block program would consist of quickly 
delivering a small number of operational aircraft capable of 
meeting at least 80% of mission requirements. As the 
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aircraft are used by operational units, additional feedback 
would be inputted to improve the performance of subsequent 
production blocks of aircraft. Operational input to the 
test program would insure that the aircraft, as designed, 
are in fact meeting real needs. This would allow contrac- 
tors to continue utilizing test aircraft to solve any 
problems which inhibited the attainment of all performance 
guaranties, without delaying the delivery of operational 
aircraft to DOD. Although this would most certainly result 
in a requirement to retrofit/modify the initial block of 
aircraft, it is felt that this procedure would ultimately 
provide a more capable aircraft to operational units in a 
more timely manner. However, this strategy would create a 
potential problem in the upgrading of the early delivered 
aircraft. Usually, the cost of modification is significant- 
ly greater than the cost of initial production. 

10 . Joint Contractor/Military Testing 

Various degrees of joint testing have been prac- 
ticed by DOD. It consists of combining the efforts of the 
contractors and the military in the parallel testing. These 
combined efforts include the development, test, and evalua- 
tion phases. Advantages of this approach include: 

1. Early involvement of operational aircrew and sharing 
of test data from the genesis of the test program 

2. Reduced duplication of efforts 

3. A new critical perspective of the progress of the 
program that could help identify shortcomings or 
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problem areas in the early stages of the test program 
that otherwise might not have been identified until 
the Operational Test and Evaluation phase; this would 
lead to substantial savings in time and costs. 

Contractors feel that they no longer have primary 
control over their own flight test programs. The require- 
ment to use government test sites and facilities, which are 
inadequately equipped to support simultaneous test programs 
of major aircraft systems, is viewed as counterproductive. 
These inadequacies require that program priorities be 
established which results in test delays and some, if not 
all, of the following ramifications to the contractors: 

1. They have to relocate their personnel and test 
equipment to governmental facilities 

2. They have to train military test pilots in highly 
instrumented test aircraft 

3. Flight schedules must be submitted to the military for 
approval before actual real time planning can begin 

4. They are subject to DOD rules and regulations, which 
many find too restrictive 

5. Their schedules are often reviewed by military 
personnel who may not be familiar with the intricacies 
of the contractor's flight test program. The explana- 
tions and briefings required to clear up these 
misunderstandings cause unnecessary interruptions in 
the test program 

6. The turnover of key military personnel during the Test 
and Evaluation programs often precipitates revisions 
of previously reviewed and approved cockpit/aircraft 
configurations . 

From the contractors' viewpoint, the requirements 
specifications initially established at the beginning of the 
program are constantly modified during the System Test and 
Evaluation life cycle. Contractors contend that, given 
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accurate r detailed, and complete DOD specifications, they 
could complete the required Developmental Test and Evalua- 
tion more effectively and efficiently without interference 
from external agencies. As such, the contractors would 
ultimately be responsible for development testing, test 
management, and more important, test success. 

It is important to note that opinions in this issue 
have been extremely bimodal. Historical data could be found 
to support both sides of the issue as both management 
strategies have been successful in producing high quality 
aircraft for the Department of Defense. Since these 
arguments cover only a partial view of the entire problem, 
further studies are necessary to determine which strategy is 
capable of producing the highest quality and most cost- 
efficient aircraft. Specifically, a more integral approach 
appears to be the only way to assess the total impact of the 
two strategies. 

11 . Political 

Last but not least. Congressional funding delays 
are cited as having a non-trivial impact on the total system 
cost. These delays caused an unnecessary time lag between 
the development and production phases. This lack of im- 
mediate follow-up results in loss of experience and exper- 
tise gained in the prototyping phase and serious economical 
and personnel problems in maintaining qualified manpower. 



78 



C . SUMMARY 

The interviews conducted with Defense Contractors and 
the military test centers resulted in numerous recommen- 
dations for improvement of the present process of System 
Test and Evaluation. While the following summarized list of 
recommendations addressed in this chapter represents neither 
an exhaustive nor final list of possible actions, they do 
highlight several areas of major concern; 

1. The use of simulation in software testing 

2. The utilization of a structured analysis and design 
methodology to develop software 

3. The implementation of distributed systems using 
parallel processors 

4. The development of more efficient power units, 
thereby reducing the total weight and volume of 
avionics in the aircraft 

5. The reduction of the required number of test 
aircraft 

6. The delivery of aircraft in a block (phase) program. 
Also, further studies are necessary to determine if joint 
testing is producing the highest quality and most 
cost-efficient aircraft. 
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APPENDIX A: CONTRACTOR FORM LETTER #1 



Naval Postgraduate School 
Monterey# Ca. 93943 
October 24# 1986 



Dear 



I have extracted some information from our thesis proposal 
to give you a better idea of what this project entails and will 
try to accomplish. I have also enclosed a tentative format 
depicting a Work Breakdown Structure/Functional Cost Element 
Matrix which, with some flexibility# will attempt to use in our 
data collection effort. I will call you on the rd to answer 
any questions you may have concerning the project and provide you 
an update of our latest discussions with NAVAIR. 



The week of November 16th# we will be making a cross-country 
collection trip to all of the defense contractor facilities 
involved in this project. We would like to meet with you in 
Columbus, Monday morning, November 17th, to collect the initial 
data you have extracted# and discuss the project in more detail. 
We realize that doesn't give you very much time# but we are 
trying to have most of the initial data# from all sources, 
collected and reviewed for discussion at a NAVAIR meeting in 
Washington D, C. November 20th. This meeting will be used to 
finalize the focus of the data needed to complete this research 
project. We will contact you immediately upon our return from 
the NAVAIR meeting to advise you of the results of the meeting 
and to eliminate any ambiguities concerning the data we received 
and finalize our further data collection requirements. We 
anticipate two follow-on trips. The first for final collection 
of data# the second to present the results of our research and 
copies of our cost estimating tools. 



The basic data we will need will be the direct labor hours 
and flight hours related to all DOD aircraft systems your 
facility helped produce. 
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The information we require necessitates a breakdown of 
Contractor Flight Test, Wind Tunnel Test, Static Test, and 
Fatigue Test into the lowest available indentured sub-elements 
that your company used during system test and evaluation. 

It is important to our efforts that we obtain the Contractor 
Flight Test labor hours, in particular, in a time-phased fashion 
along with time-phased flight hours, so as to be able to relate 
the two. We would also like to obtain time-phased cost data for 
the other test elements if possible. 

We would like to meet with the individuals who develop the 
initial Work Breakdown Structure (WBS) estimates to discuss their 
ideas on what factors became the cost-drivers. If available, 
inputs from flight test engineers would contribute significantly 
to the balance of the data we are collecting. 

I realize that we are asking for what you may consider to 
be part of your company's proprietary information. I assure you, 
in the strongest possible terms, that all data will be held in 
the strictest confidence, used only for the generation of a cost 
estimating algorithm, and not divulged outside of the government. 



We appreciate your cooperation and look forward to working 
together in the coming months. 



Semper Fi , 



William J. Foster 



David F. Lee 
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A. PROJECT TEAM 



Dr. Tung X. Bui Assistant Professor NPS 
LCDR Robert W. Fonnesbeck 529-66-1259 USN 
Major William J. Foster 537-50-0275 USMC 
Major David J. Lee 568-74-7785 USMC 
LT Walter J. Moore 273-42-9741 USN 

Com: 408-646-2995/2630 

Av; 878-2995/2630 

B. DISCUSSION 

Current budget constraints have resulted in an increasing 
emphasis on the cost effective procurement of aircraft 
systems within the Department of Defense. Of late, the Cost 
Analysis Division of the Naval Air Systems Command 
(NAVAIR) has received numerous requests to provide explicit 
estimates for the sub-elements of aircraft system test and 
evaluation efforts. These data represent lower levels in 
the Work Breakdown Structure than previously required. This 
additional data and the methods necessary to produce them 
are needed in order to estimate aircraft system test and 
evaluation costs with better accuracy and in greater detail 
than is currently feasible. 

NAVAIR has funded this project to collect the data and 
develop an accurate and reliable way to estimate these cost 
relationships. The key problems related to this research 
will involve the data collection from the defense 
contractors and the standardization of their data. 
Representatives at NAS Patuxent River, Md., and Edwards Air 
Force Base Flight Test Center have already stated that they 
will assist in the collection of data at their 
installations . 



C. AREA OF RESEARCH 

Estimation of cost drivers in Aircraft System Test and 
Evaluation (ASTE); conceptual modeling and some empirical 
evidence. (Air Force, Navy, and civilian contractors — 
subject to data availability) 



D. RESEARCH QUESTIONS 

1. What are the cost drivers in ASTE? 

2. How do the characteristics and complexity of the 
aircraft relate to the average engineering hour? 
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3. What statistical and econometric models can be 
appropriately applied to ASTE? Is multiple regression 
appropriate to this problem? 

4. Are the military work-breakdown structures pertinent to 
ASTE? 

5. How can a computer system (hardware and software) be 
designed and implemented to conduct testing of the 
models? 

6. How have data been defined and collected in different 
test sites (military and civilian)? Can they be 
filtered, and standardized? What extrapolation method 
should be used for missing data? 

7. How do these cost drivers affect the average 
engineering hour per hour of flight? 

8. Is there any correlation between the number of 
instrumentation channels and the number of engineering 
hours? 

9. Is there a linear cost relationship between different 
types of aircraft with respect to their complexity and 
characteristics? 

10. How sensible and stable are the elasticities of the 
cost drivers? 

11. What are the implications of the findings? 

12. Where is it most cost-effective to conduct future 
military ASTE — at the contractors facilities or at the 
established flight test centers? Is the answer the 
same for both flight test centers? 

13. Do the findings have an impact on the service's 
operational test and evaluation? 

14. How does inflation affect the predictive power of the 
findings? 



E. SCOPE OF THE RESEARCH 

Develop essentially parametric cost estimating relationships 
for selected system test and evaluation elements and each 
appropriate functional cost category by: 

1. Defining a sample of pertinent aircraft problems. 
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2. Formulating a work break-down structure/functional cost 
elements matrix. 

3. Acquiring historical cost data. 

4. Organizing the data for analysis. 

5. Employing statistical methods to develop cost 
estimating relationships. 

6. Documenting data, sources, rationale, and methodology. 



F . METHODOLOGY 

This research seeks to apply microeconomic theory and cost 
analysis in the area of Aircraft Systems Test and Evaluation 
(ASTE). Multiple linear and logarithmic regression are 
considered as analytical tools. This research also applies 
the concepts defined in system analysis and design to 
implement a decision support software for ASTE. 

Interviews with defense contractors, Patuxent River Naval 
Test Flight Center, Edwards Air Force Base Flight Test 
Center, and on-site data collection are envisaged for the 
first part of the research. Utilization of statistical 
software packages to identify elasticities of cost drivers 
will constitute the second part of the research. 



I. BENEFITS OF STUDY 

There is an increasing requirement from high levels that the 
DOD's aircraft cost estimators/analysts provide explicit 
estimates for the sub-elements of aircraft systems test and 
evaluation efforts. This research project will attempt to 
develop cost estimating relationships for test and 
evaluation elements used for aircraft systems. The results 
of this project will benefit DOD, Patuxent River Naval Test 
Flight Center, Edwards Air Force Base Flight Test Center, 
and civilian contractors. 
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Ihis fori contains the inforiation requested. Any further breakdonn 
if available, is desirable. 



iS 

M cn 





«/) c 








*/• 



(_a 






— fx 






85 



Nuaber of Iliqht test A/C and first flight of each, 
flight hours (tfae phased). 

Mind tunnel occupancy hours ftlie phased), 
fype of Nind tunnel test. 

Nuiber of Instruientation channels for instruient flight test A/C. 



APPENDIX B: NAVAIR PROJECT POINTS OF CONTACT 



Air Force Flight Test Center 
Mrs. Carol Gutherie 
Code AC 

Edwards Air Force Base, Ca. 93523 

Boeing Aerospace Company 
Mr. Henry F. Schwartz 
M/S 1758 
P.O. Box 3707 
Seattle, Wa. 98124 

Douglas Aircraft Company 
Mr. Len Chartier 
Cl-243 18A-40 
3855 Lakewood Blvd. 

Long Beach, Ca. 90846 

Douglas Aircraft Company 
Mr . Loren Frye 
Cl-240 41-80 
3855 Lakewood Blvd. 

Long Beach, Ca. 90846 

Fairchild Republic Company 
Mr. Stan Granowetter 
Conklin Street 
Farmingdale, N.Y. 11735 

Fairchild Republic Company 
Mr. Robert Hoffman 
Building 55 
Conklin Street 
Farmingdale, N.Y. 11735 

General Dynamics Corporation 

Mr. John Denheyer 

M/Z 1638 

P.O. Box 748 

Fort Worth, Tx. 76101 

Grumman Aerospace Corporation 
Mr. Jim McDonagh 
MS C20-GHQ 
Bethpage, N.Y. 11714 



805-227-3710 



206-655-3244 



213-593-0920 



213-593-4745 



516-531-3251 



516-531-2752 



817-777-1581 



516-575-5158 
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404-424-2741 



Lockheed-Georgia Company 
Mr . Bob Farmer 
Dept. 81-11, Zone 23 
86 South Cobb Dr. 

Marietta, Ga. 30063 

Lockheed-Calif ornia Company 
Mr. Phil Finkle 
Dept. 81-16, B/66, P/A-1 
P.O. Box 551 
Burbank, Ca. 91520 

LTV Aerospace and Defense 

Aero Products Division 

Mr. Clint Miller 

MS 220-70 

P.O. Box 225907 

Dallas, Tx. 75265 

McDonnell Aircraft Company 

Mr . Raymond Yarck 

Dept. 015 

P.O. Box 516 

St. Louis, Mo. 63166 

Naval Air Systems Command 
Mr. Bob Patterson 
AIR-5243 

Jefferson Plaza 2 
Washington D.C. 20361 

Naval Air Test Center 
Karen Tyson 

Business Resource Department 
Antisubmarine Aircraft Test Directorate 
Patuxent River, Md. 20670-5304 

Rockwell International Corporation 
Mr. Gayle Turner 
Bldg. 3 

4300 E. Fifth Ave. 

Columbus, Oh. 43216 

Rockwell International Corporation 

Mr. George Bostater 

Dept 196, 011-ZR09 

P.O. Box 90098 

Los Angeles, Ca. 90009 



818-847-3343 



214-266-8755 



314-234-7100 



202-692-3935 



301-863-3691 



614-239-2796 



213-414-4819 
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APPENDIX 3; CONTRACTOR FORM LETTER #2 



December 5, 1986 



Navair Research Team 
Code 54Bd 

Naval Postgraduate School 
Monterey, Ca. 93943 



We would like to thank you again for the opportunity we had 
to meet with you and visit your facilities to discuss our 
research project last month. Your inputs were extremely valuable 

in giving us an overview of 's methods used for the cost 

estimation of aircraft systems and your test and evaluation 
cprograms. At that time, we promised to clarify our data 
collection requirements following a meeting with our Naval Air 
Systems Command (NAVAIR) project sponsor on 20-21 November. The 
information you provided was discussed and helped focus the scope 
of our project and the specific areas of data required for our 
research . 

The scope of this project will initially include only the 
following areas of aircraft system test and evaluation; 

Contractor Flight Test 
Wind Tunnel and Article Test 
Static Article and Test 
Fatigue Article and Test. 

In our initial letter, we provided a matrix of Work Breakdown 
Structure (WBS) sub-elements in these areas and a listing of the 
following direct labor hour/dollar cost categories pertaining to 
these sub-elements; 

Engineering Hours 
Manufacturing Hours 
Tooling Hours 
Quality Control Hours 
Logistic/Product Support Hours 
Material Dollars 
Total Dollars. 

The basic System Test and Evaluation data required for our 

project are direct labor hours and dollar values for the 

aircraft system(s) your facility produced, broken down into the 
sub-element matrix, and also listed in a time-phased fashion. 

In order to normalize this information between contractors, 
it is important that we know your specific definition of the WBS 
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sub-elements used by your company for direct labor hour tracking. 
For the direct labor hours, we request that you provide a brief 
description of how the engineering, manufacturing, tooling, 
quality control and support hour categories were derived if your 
records differ from the Contractor Cost Data Reporting methods. 

For the aircraft flight test portion of our research, we 
request the number of aircraft used in your flight test program 
with associated flight hours. 

We request the data for direct labor hour/dollar categories 
and the flight test hours be expressed in a time-phased fashion. 
If this data is not available from your records time-phased, 
please estimate the start and completion dates and indicate the 
peak activity levels within the test period. 

A cost driver indicated for the flight test program was the 
total number of separate tests to be conducted during the flight 
phase. If you have information available to indicate the total 
number of separate tests and a brief description of these test 
categories, it will help in the quality of our final product. 

We feel that this research can be valuable to our military 

project sponsor and also useful to as a cost 

estimating tool. The accuracy of our final product will depend 
primarily on the data we receive from your company and our 
ability to correlate it with the data from the other contractors 
we have visited. 

We realize the constraints of your busy schedule, but would 
appreciate to be able to receive as much of this data as possible 
by January 10, 1987 to be used in a meeting which will be held 
with the NAVAIR project sponsor January 15, 16 and 17. 

We again express our thanks for the assistance that you have 
provided and your continued interest in our program. We will 
call you next week and attempt to answer questions that you may 
have concerning this data format or limitations that may be 
peculiar to a specific aircraft system. 



Sincerely, 
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The following are to be broken down by; 

1 . Engineering hours 

2. Manufacturing hours 

3. Tooling hours 

4. QC/ILS/Product support 

5. Material $ 

6. Total $ 

CONTRACTOR FLIGHT TEST 
INSTRUMENTATION 
FLIGHT AND DEMONSTRATION 
FUNCTIONAL GROUND CHECKOUT 
FLIGHT TEST AND DEMONSTRATION 
FLIGHT TEST SUPPORT 

FLIGHT TEST SPARES & REPAIR PARTS 
SUPPORT EQUIPMENT 

CONTRACTOR TECHNICAL/MAINTENANCE SERVICES 

WIND TUNNEL ARTICLE & TEST 
WIND TUNNEL ARTICLE 
WIND TUNNEL TEST 

STATIC ARTICLE & TEST 
STATIC ARTICLE 
STATIC TEST 

FATIGUE ARTICLE & TEST 
FATIGUE ARTICLE 
FATIGUE TEST 



Other breakdowns ; 

1. # of flight test a/c and first flight of each. 

2. Flight hours ( time phased) 

3. Wind tunnel occupancy hours ( time phased) 

4. Type of wind tunnel test 

5. # of instrumentation channels for instrument flight test 
A/C. 
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